{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "de765f16-341b-4dc3-b59d-b16437c7e050",
   "metadata": {},
   "source": [
    "\n",
    "# LanceDB \n",
    "\n",
    "https://lancedb.com/\n",
    "\n",
    "LanceDB is a developer-friendly, open source vector database for multi-modal AI with zero management overhead. Installs in seconds and scales to billions of embeddings at a fraction of the cost of other vector databases. You can even stream data directly from object storage for training or fine-tuning.\n",
    "\n",
    "https://lancedb.github.io/lancedb/\n",
    "\n",
    "\n",
    "We use LanceDB as a vector database to embed the following contents\n",
    "\n",
    "1. Structurizr example codes cloned from github\n",
    "\n",
    "    https://github.com/structurizr/examples.git\n",
    "    https://github.com/structurizr/dsl.git\n",
    "\n",
    "Please refer to the following folders:\n",
    "\n",
    "    structurizr/dsl/examples\n",
    "    structurizr/dsl/docs/cookbook\n",
    "\n",
    "2. structurizr language reference markdown document cloned from github\n",
    "\n",
    "    https://github.com/structurizr/dsl.git\n",
    "   \n",
    "Please refer to `structurizr/dsl/docs/language-reference.md`\n",
    "\n",
    "Please refer to `README-MarkdownLoader.md` for environment setup to process markdown documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e25cdbdc-daee-4a9b-b363-9080dc216927",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: langchain\n",
      "Version: 0.1.9\n",
      "Summary: Building applications with LLMs through composability\n",
      "Home-page: https://github.com/langchain-ai/langchain\n",
      "Author: \n",
      "Author-email: \n",
      "License: MIT\n",
      "Location: /opt/conda/lib/python3.11/site-packages\n",
      "Requires: aiohttp, dataclasses-json, jsonpatch, langchain-community, langchain-core, langsmith, numpy, pydantic, PyYAML, requests, SQLAlchemy, tenacity\n",
      "Required-by: \n"
     ]
    }
   ],
   "source": [
    "!pip show langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "aa827cda-eb15-4767-9f8a-92d18bc9840c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: lancedb\n",
      "Version: 0.6.2\n",
      "Summary: lancedb\n",
      "Home-page: \n",
      "Author: \n",
      "Author-email: LanceDB Devs <dev@lancedb.com>\n",
      "License: Apache-2.0\n",
      "Location: /opt/conda/lib/python3.11/site-packages\n",
      "Requires: attrs, cachetools, click, deprecation, overrides, pydantic, pylance, pyyaml, ratelimiter, requests, retry, semver, tqdm\n",
      "Required-by: \n"
     ]
    }
   ],
   "source": [
    "!pip show lancedb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e07b2c7d-62d7-457c-bbcb-298b29aa522c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please define the following env variables in the .env file \n",
    "# \n",
    "# ollama_url = http://ollama:11434\n",
    "# model_name = mistral:instruct\n",
    "# embedding_model=nomic-embed-text\n",
    "# "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fa2048ea-dbc4-49d7-b855-8b546dce2edb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the utility.py to get llm, embedding and output parser\n",
    "import utility\n",
    "embeddings = utility.get_embeddings()\n",
    "llm = utility.get_llm()\n",
    "output_parser = utility.get_output_parser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13a9da9e-5b71-46c5-b392-edd84f06ed23",
   "metadata": {},
   "source": [
    "## Create LanceDB table with embedding model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f7255077-07b3-40f4-952a-0ba904d3e08c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2024-03-16T02:13:26Z WARN  lance::dataset] No existing dataset at /home/jovyan/work/lancedb/example.lance, it will be created\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 250 ms, sys: 31.5 ms, total: 282 ms\n",
      "Wall time: 939 ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2024-03-16T02:13:26Z WARN  lance::dataset] No existing dataset at /home/jovyan/work/lancedb/language.lance, it will be created\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# https://lancedb.github.io/lancedb/guides/tables/#creating-empty-table\n",
    "import pyarrow as pa\n",
    "import lancedb\n",
    "\n",
    "db = lancedb.connect(\"lancedb\")\n",
    "\n",
    "# schema = pa.schema(\n",
    "#     [\n",
    "#         pa.field(\"vector\", pa.list_(pa.float32(), list_size=768)),\n",
    "#         pa.field(\"text\", pa.string())\n",
    "#     ]\n",
    "# )\n",
    "# table = db.create_table(\"structurizr\", schema=schema, mode = \"overwrite\")\n",
    "## a folder \"lancedb/structurizr.lance\" will be created automatically\n",
    "\n",
    "# table to store code examples\n",
    "example = db.create_table(\"example\", data = [{\"vector\":embeddings.embed_query(\"example\"), \"text\":\"example\"}], mode = \"overwrite\")\n",
    "# table to store language reference\n",
    "language = db.create_table(\"language\", data = [{\"vector\":embeddings.embed_query(\"language\"), \"text\":\"language\"}], mode = \"overwrite\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b24f2cde-14e3-4fb8-9ade-05b97c0beee3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              vector     text\n",
      "0  [0.86832356, 0.6491402, -3.231392, -0.9464089,...  example\n"
     ]
    }
   ],
   "source": [
    "# Get the updated table as a pandas DataFrame\n",
    "print(example.to_pandas())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "039a827a-595a-4ecc-ae4c-9d4a0ae25397",
   "metadata": {},
   "source": [
    "## Embeb Local Knowledage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b3bef9c4-29f5-473a-a400-d425ecc3b92f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make up some fact to demonstrate RAG on knowledage base\n",
    "\n",
    "knowledage = \"Some famous companies such as ABC and XYZ are using Structurizr DSL for software architecture modeling and documentation.\"\n",
    "\n",
    "example.add(data = [{\"vector\":embeddings.embed_query(knowledage), \"text\":knowledage}])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3d426994-e34d-48d6-b994-bc62f8845b14",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.65 ms, sys: 5.81 ms, total: 7.46 ms\n",
      "Wall time: 65.7 ms\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vector</th>\n",
       "      <th>text</th>\n",
       "      <th>_distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[-0.14687483, 1.3784066, -2.4142506, 0.1442473...</td>\n",
       "      <td>Some famous companies such as ABC and XYZ are ...</td>\n",
       "      <td>398.858582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[0.86832356, 0.6491402, -3.231392, -0.9464089,...</td>\n",
       "      <td>example</td>\n",
       "      <td>669.272583</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              vector  \\\n",
       "0  [-0.14687483, 1.3784066, -2.4142506, 0.1442473...   \n",
       "1  [0.86832356, 0.6491402, -3.231392, -0.9464089,...   \n",
       "\n",
       "                                                text   _distance  \n",
       "0  Some famous companies such as ABC and XYZ are ...  398.858582  \n",
       "1                                            example  669.272583  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "example.search(embeddings.embed_query(\"Company ABC\")) \\\n",
    "    .limit(5) \\\n",
    "    .nprobes(20) \\\n",
    "    .refine_factor(10) \\\n",
    "    .to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "11f65c41-fa12-406f-b357-225c5c837299",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.68 ms, sys: 1.91 ms, total: 5.59 ms\n",
      "Wall time: 74 ms\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>vector</th>\n",
       "      <th>text</th>\n",
       "      <th>_distance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[-0.14687483, 1.3784066, -2.4142506, 0.1442473...</td>\n",
       "      <td>Some famous companies such as ABC and XYZ are ...</td>\n",
       "      <td>364.171967</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[0.86832356, 0.6491402, -3.231392, -0.9464089,...</td>\n",
       "      <td>example</td>\n",
       "      <td>757.671997</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              vector  \\\n",
       "0  [-0.14687483, 1.3784066, -2.4142506, 0.1442473...   \n",
       "1  [0.86832356, 0.6491402, -3.231392, -0.9464089,...   \n",
       "\n",
       "                                                text   _distance  \n",
       "0  Some famous companies such as ABC and XYZ are ...  364.171967  \n",
       "1                                            example  757.671997  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "example.search(embeddings.embed_query(\"software architecture\")) \\\n",
    "    .limit(5) \\\n",
    "    .nprobes(20) \\\n",
    "    .refine_factor(10) \\\n",
    "    .to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b5ecd71-1e28-4f73-8160-b59e889a2a9d",
   "metadata": {},
   "source": [
    "## Create vectorstore backed by LanceDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5cc644e6-61a2-4df7-b363-816f0db49856",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.77 ms, sys: 992 µs, total: 3.77 ms\n",
      "Wall time: 3.42 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# embed documents into vectorstore\n",
    "from langchain_community.vectorstores import LanceDB\n",
    "\n",
    "# vectorstore = LanceDB.from_documents(docs,embeddings,connection=table)\n",
    "example_vectorstore = LanceDB(example, embeddings)\n",
    "language_vectorstore = LanceDB(language, embeddings)\n",
    "\n",
    "# add texts to vectorstore\n",
    "# vectorstore.add_texts(['text1', 'text2'])\n",
    "# result = vectorstore.similarity_search('text1')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a6b0217-d3b2-4b2c-9544-162e3ef40495",
   "metadata": {},
   "source": [
    "## Similarity search in vectorstore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "312e95d1-141e-4f88-b109-362392f4d4e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*******\n",
      "Some famous companies such as ABC and XYZ are using Structurizr DSL for software architecture modeling and documentation.\n",
      "\n",
      "*******\n",
      "example\n",
      "\n",
      "CPU times: user 5.38 ms, sys: 1.23 ms, total: 6.6 ms\n",
      "Wall time: 41.6 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "result = example_vectorstore.similarity_search(\"Company ABC\")\n",
    "for doc in result:\n",
    "    print(f'*******\\n{doc.page_content}\\n', flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eae1cf07-bfc4-4979-8ba8-de9136f528d3",
   "metadata": {},
   "source": [
    "## Load Structurizr DSL sample codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a11baec0-71af-46f8-bb39-bacc1c354c55",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 98%|█████████▊| 54/55 [00:00<00:00, 3862.62it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 70.9 ms, sys: 24.9 ms, total: 95.8 ms\n",
      "Wall time: 152 ms\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Load the dsl file from local machine for embedding\n",
    "# https://python.langchain.com/docs/modules/data_connection/document_loaders/file_directory\n",
    "# https://python.langchain.com/docs/integrations/document_loaders/unstructured_file\n",
    "from langchain_community.document_loaders import DirectoryLoader, TextLoader\n",
    "\n",
    "# load examples dsl\n",
    "loader = DirectoryLoader('structurizr/dsl', glob=\"**/*.dsl\", show_progress=True, use_multithreading=True, loader_cls=TextLoader)\n",
    "\n",
    "docs1 = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "97c4e0ee-d1b7-4f11-914c-4b19d7812b06",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='workspace {\\n\\n    model {\\n        u = person \"User\"\\n        s = softwareSystem \"Software System\" {\\n            webapp = container \"Web Application\" \"\" \"Spring Boot\"\\n            database = container \"Database\" \"\" \"Relational database schema\"\\n        }\\n\\n        u -> webapp \"Uses\"\\n        webapp -> database \"Reads from and writes to\"\\n        \\n        development = deploymentEnvironment \"Development\" {\\n            deploymentNode \"Developer Laptop\" {\\n                containerInstance webapp\\n                deploymentNode \"MySQL\" {\\n                    containerInstance database\\n                }\\n            }\\n        }\\n    }\\n\\n    views {\\n        deployment * development {\\n            include *\\n            autoLayout lr\\n        }\\n    }\\n    \\n}' metadata={'source': 'structurizr/dsl/docs/cookbook/deployment-view/example-1.dsl'}\n"
     ]
    }
   ],
   "source": [
    "# print the loaded documents, the .dsl content has been loaded as the page_content\n",
    "print(docs1[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fa118774-c93e-41b2-b903-0994bc9eaaed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='workspace {\\n\\n    model {\\n        u = person \"User\"\\n        s = softwareSystem \"Software System\"\\n\\n        u -> s \"Uses\"\\n    }\\n\\n    views {\\n        systemContext s {\\n            include *\\n            autoLayout lr\\n        }\\n    }\\n    \\n}' metadata={'source': 'structurizr/dsl/docs/cookbook/system-context-view/example-1.dsl'}\n"
     ]
    }
   ],
   "source": [
    "print(docs1[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "73852a42-1f36-4de9-8737-c68bb9def2ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Split the document content into small chunks of texts for embedding\n",
    "# from langchain.text_splitter import CharacterTextSplitter, RecursiveCharacterTextSplitter\n",
    "# # text_splitter1 = CharacterTextSplitter(chunk_size=300, chunk_overlap=50)\n",
    "# text_splitter1 = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50, separators=[\"\\n\\n\", \"\\n\",\" \",\"\"])\n",
    "# docs1 = text_splitter1.split_documents(docs1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb2e8014-e551-40e3-8257-096128d8ac55",
   "metadata": {},
   "source": [
    "## Embed Structurizr DSL sample codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7e218ab7-b373-4158-91ba-f58d07a97533",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 91.6 ms, sys: 20.6 ms, total: 112 ms\n",
      "Wall time: 36.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# vectorstore.add_texts(['text1', 'text2'])\n",
    "result = example_vectorstore.add_documents(docs1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0fb636c-7ef7-4f59-8ea0-68dccba066c4",
   "metadata": {},
   "source": [
    "## Load markdown document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "ab3296e7-e2ab-4026-bd41-49d6b0b25230",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.11 s, sys: 191 ms, total: 1.3 s\n",
      "Wall time: 2.78 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# only load structurizr/dsl/docs/language-reference.md\n",
    "# https://python.langchain.com/docs/modules/data_connection/document_loaders/markdown\n",
    "# pip install unstructured\n",
    "from langchain_community.document_loaders import UnstructuredMarkdownLoader\n",
    "loader = UnstructuredMarkdownLoader(\"structurizr/dsl/docs/language-reference.md\", mode=\"elements\")\n",
    "\n",
    "docs2 = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "77291c6e-9840-464c-83f7-58d3b3037e88",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Language reference' metadata={'source': 'structurizr/dsl/docs/language-reference.md', 'last_modified': '2023-07-08T04:23:38', 'page_number': 1, 'languages': ['eng'], 'filetype': 'text/markdown', 'file_directory': 'structurizr/dsl/docs', 'filename': 'language-reference.md', 'category': 'Title'}\n"
     ]
    }
   ],
   "source": [
    "print(docs2[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "78103587-fa1a-4b4e-9439-c5f6fa2b0f59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='The Structurizr DSL provides a way to define a software architecture model\\n(based upon the C4 model) as text, using a domain specific language (DSL).\\nThe DSL is rendering tool independent, and to render diagrams you will need to use one of the tools listed at\\nStructurizr DSL - Rendering tools.' metadata={'source': 'structurizr/dsl/docs/language-reference.md', 'last_modified': '2023-07-08T04:23:38', 'page_number': 1, 'languages': ['eng'], 'parent_id': '3033ec4f49f78020d6ced9cea119fa0e', 'filetype': 'text/markdown', 'file_directory': 'structurizr/dsl/docs', 'filename': 'language-reference.md', 'category': 'NarrativeText'}\n"
     ]
    }
   ],
   "source": [
    "print(docs2[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4f58036e-78c7-455f-9d97-20424e98d4b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='Please see the DSL cookbook for a tutorial guide to the Structurizr DSL.' metadata={'source': 'structurizr/dsl/docs/language-reference.md', 'last_modified': '2023-07-08T04:23:38', 'page_number': 1, 'languages': ['eng'], 'parent_id': '3033ec4f49f78020d6ced9cea119fa0e', 'filetype': 'text/markdown', 'file_directory': 'structurizr/dsl/docs', 'filename': 'language-reference.md', 'category': 'NarrativeText'}\n"
     ]
    }
   ],
   "source": [
    "print(docs2[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "03f6a651-e1d1-4257-9f04-80abc5165b4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# from langchain.text_splitter import MarkdownTextSplitter\n",
    "# text_splitter2 = MarkdownTextSplitter(chunk_size=200, chunk_overlap=0)\n",
    "# docs2 = text_splitter2.split_documents(docs2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "35b12934-e2e0-4738-bb92-d4efd075971a",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "##https://python.langchain.com/docs/modules/data_connection/document_transformers/code_splitter#markdown\n",
    "# from langchain.text_splitter import Language,RecursiveCharacterTextSplitter\n",
    "\n",
    "# md_splitter = RecursiveCharacterTextSplitter.from_language(\n",
    "#     language=Language.MARKDOWN, chunk_size=200, chunk_overlap=0\n",
    "# )\n",
    "\n",
    "# md_docs = md_splitter.split_documents(mddoc)\n",
    "# print(len(md_docs))\n",
    "# #730\n",
    "\n",
    "# #This is a long document we can split up.\n",
    "# with open(\"structurizr/dsl/docs/language-reference.md\") as f:\n",
    "#     content = f.read()    \n",
    "# md_docs = md_splitter.create_documents([content])\n",
    "# print(len(md_docs))\n",
    "# #444"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "96e57f0c-e2d5-4891-a53c-6af555501f91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(md_docs[2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "34ec4333-0598-4cb9-817d-1982d143195f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# print(md_docs[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc6c312d-49ab-4285-90f3-c6eef35148b4",
   "metadata": {},
   "source": [
    "## Embed markdown document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "fe8ed53d-18fd-473a-bb4f-6ff79d994b76",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start at    ... 2024-03-16 02:16:53.846002\n",
      "\n",
      "Complete at ... 2024-03-16 02:17:40.604671\n",
      "Total time used 46.755868434906006 seconds\n",
      "\n",
      "CPU times: user 774 ms, sys: 132 ms, total: 905 ms\n",
      "Wall time: 46.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "time1 = utility.chain_starts()\n",
    "\n",
    "result = language_vectorstore.add_documents(docs2)\n",
    "\n",
    "utility.chain_completes(time1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f9dad82b-122c-4f86-99b4-f2acb1c674c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import time, datetime\n",
    "# # embed documents into vectorstore\n",
    "# from langchain_community.vectorstores import LanceDB\n",
    "\n",
    "# # print(f\"Start embeding    ... {datetime.datetime.now()}\",flush=True)\n",
    "# # time1=time.time()\n",
    "\n",
    "# time1 = utility.chain_starts()\n",
    "\n",
    "# vectorstore = LanceDB.from_documents(docs,embeddings,connection=table)\n",
    "\n",
    "# utility.chain_completes(time1)\n",
    "\n",
    "# time2=time.time()\n",
    "# print(f\"Complete embeding ... {datetime.datetime.now()}\",flush=True)\n",
    "# print(f'Total time used {time2 - time1} seconds')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "fbaa621c-eaf2-4358-b544-1ebeb4617028",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 1: Add a list of dicts to a table\n",
    "# data = [{\"vector\": embeddings.embed_query(texts2[2].page_content), \"text\": texts2[2].page_content},\n",
    "#         {\"vector\": embeddings.embed_query(texts2[3].page_content), \"text\": texts2[3].page_content}]\n",
    "\n",
    "# table.add(data)\n",
    "\n",
    "# db = lancedb.connect('lancedb')\n",
    "# table = db.open_table('my_table')\n",
    "# vectorstore = LanceDB(table, embeddings)\n",
    "# vectorstore.add_texts(['text1', 'text2'])\n",
    "# result = vectorstore.similarity_search('text1')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "490ef88d-65fc-4f4e-b29a-1feef185815b",
   "metadata": {},
   "source": [
    "## Sample search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "02b3f72b-a7e1-48f9-8a08-e61399644775",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*******\n",
      "The person keyword defines a person (e.g. a user, actor, role, or persona).\n",
      "\n",
      "*******\n",
      "person\n",
      "\n",
      "*******\n",
      "Person\n",
      "\n",
      "*******\n",
      "person\n",
      "\n",
      "CPU times: user 11.7 ms, sys: 5.2 ms, total: 16.9 ms\n",
      "Wall time: 96.5 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "results = language_vectorstore.similarity_search('keyword to define a person')\n",
    "\n",
    "for doc in results:\n",
    "    print(f'*******\\n{doc.page_content}\\n', flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "ad0b1f98-73dc-4536-9b03-5e08c837f0a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*******\n",
      "The person keyword defines a person (e.g. a user, actor, role, or persona).\n",
      "\n",
      "*******\n",
      "person\n",
      "\n",
      "*******\n",
      "person\n",
      "\n",
      "*******\n",
      "Person\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for doc in results:\n",
    "    print(f'*******\\n{doc.page_content}\\n', flush=True)\n",
    "# print(results[0].page_content, flush=True)\n",
    "# print(len(results))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a4355e8-7c7d-4590-b974-3f70c93cbb22",
   "metadata": {},
   "source": [
    "\n",
    "## Indexing\n",
    "\n",
    "https://lancedb.github.io/lancedb/concepts/vector_search/\n",
    "\n",
    "https://lancedb.github.io/lancedb/concepts/index_ivfpq/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb4c0cc6-b550-43ec-93dd-558bb0ac8708",
   "metadata": {},
   "source": [
    "### Construct the index\n",
    "\n",
    "There are three key parameters to set when constructing an IVF-PQ index:\n",
    "\n",
    "- metric: Use an `L2` euclidean distance metric. We also support `dot` and `cosine` distance.\n",
    "- num_partitions: The number of partitions in the IVF portion of the index.\n",
    "- num_sub_vectors: The number of sub-vectors that will be created during Product Quantization (PQ)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0f2f34c-c46c-46f8-b3fa-c9197e0a6f53",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create and train the index for a 1536-dimensional vector\n",
    "# Make sure you have enough data in the table for an effective training step\n",
    "table.create_index(metric=\"L2\", num_partitions=1, num_sub_vectors=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37b92094-f195-4339-9fa1-a7edc71b7ce0",
   "metadata": {},
   "source": [
    "### Query the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9ff040b-bc01-4e6f-b218-b97b8c87603d",
   "metadata": {},
   "outputs": [],
   "source": [
    "table.search(embeddings.embed_query(\"Internet Banking System\")) \\\n",
    "    .limit(5) \\\n",
    "    .nprobes(20) \\\n",
    "    .refine_factor(10) \\\n",
    "    .to_pandas()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
